On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency.

A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients.

On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency.

A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both biased gradients and data inefficiency.

Knowledge distillation is a strategy of training a student network with guide of the soft output from a teacher network. It has been a successful method of model compression and knowledge transfer. However, currently knowledge distillation lacks a convincing theoretical understanding. On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency. Based on this metric, we show that for a perfect teacher, a high ratio of teacher's soft labels can be beneficial. Finally, for the case of imperfect teacher, we find that hard labels can correct teacher's wrong prediction, which explains the practice of mixing hard and soft labels.

IT and data leaders are constantly challenged to keep up with new trends in emerging and disruptive technologies, and to determine how each can best aid the organization. In the midst of all the changes going on in 2019, it gets increasingly hard to know where to invest in all this new technology. To help add clarity, here are my thoughts on some of the most important trends that will shape data management and software development for the next couple of years. The business multi-verse expands through multi-cloud as data inefficiencies are solved: Multi-cloud promises tremendous reward if it can be used properly, but data inefficiencies and complicated compliance policies hinder progress for many. Expect to see some of those data inefficiencies fade away as effective data strategies are implemented and new technologies unleash true multi-cloud functionality to the masses.

IT and data leaders are constantly challenged to keep up with new trends in emerging and disruptive technologies, and to determine how each can best aid the organization. In the midst of all the changes going on in 2019, it gets increasingly hard to know where to invest in all this new technology. To help add clarity, here are my thoughts on some of the most important trends that will shape data management and software development for the next couple of years. The business multi-verse expands through multi-cloud as data inefficiencies are solved: Multi-cloud promises tremendous reward if it can be used properly, but data inefficiencies and complicated compliance policies hinder progress for many. Expect to see some of those data inefficiencies fade away as effective data strategies are implemented and new technologies unleash true multi-cloud functionality to the masses.